60 research outputs found
Overview of the Problem List Summarization (ProbSum) 2023 Shared Task on Summarizing Patients' Active Diagnoses and Problems from Electronic Health Record Progress Notes
The BioNLP Workshop 2023 initiated the launch of a shared task on Problem
List Summarization (ProbSum) in January 2023. The aim of this shared task is to
attract future research efforts in building NLP models for real-world
diagnostic decision support applications, where a system generating relevant
and accurate diagnoses will augment the healthcare providers decision-making
process and improve the quality of care for patients. The goal for participants
is to develop models that generated a list of diagnoses and problems using
input from the daily care notes collected from the hospitalization of
critically ill patients. Eight teams submitted their final systems to the
shared task leaderboard. In this paper, we describe the tasks, datasets,
evaluation metrics, and baseline systems. Additionally, the techniques and
results of the evaluation of the different approaches tried by the
participating teams are summarized.Comment: To appear in the Proceedings of the 5th BioNLP Workshop at AC
Recommended from our members
Discovering body site and severity modifiers in clinical texts
Objective: To research computational methods for discovering body site and severity modifiers in clinical texts. Methods: We cast the task of discovering body site and severity modifiers as a relation extraction problem in the context of a supervised machine learning framework. We utilize rich linguistic features to represent the pairs of relation arguments and delegate the decision about the nature of the relationship between them to a support vector machine model. We evaluate our models using two corpora that annotate body site and severity modifiers. We also compare the model performance to a number of rule-based baselines. We conduct cross-domain portability experiments. In addition, we carry out feature ablation experiments to determine the contribution of various feature groups. Finally, we perform error analysis and report the sources of errors. Results: The performance of our method for discovering body site modifiers achieves F1 of 0.740–0.908 and our method for discovering severity modifiers achieves F1 of 0.905–0.929. Discussion Results indicate that both methods perform well on both in-domain and out-domain data, approaching the performance of human annotators. The most salient features are token and named entity features, although syntactic dependency features also contribute to the overall performance. The dominant sources of errors are infrequent patterns in the data and inability of the system to discern deeper semantic structures. Conclusions: We investigated computational methods for discovering body site and severity modifiers in clinical texts. Our best system is released open source as part of the clinical Text Analysis and Knowledge Extraction System (cTAKES)
Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning
Generative artificial intelligence (AI) is a promising direction for
augmenting clinical diagnostic decision support and reducing diagnostic errors,
a leading contributor to medical errors. To further the development of clinical
AI systems, the Diagnostic Reasoning Benchmark (DR.BENCH) was introduced as a
comprehensive generative AI framework, comprised of six tasks representing key
components in clinical reasoning. We present a comparative analysis of
in-domain versus out-of-domain language models as well as multi-task versus
single task training with a focus on the problem summarization task in DR.BENCH
(Gao et al., 2023). We demonstrate that a multi-task, clinically trained
language model outperforms its general domain counterpart by a large margin,
establishing a new state-of-the-art performance, with a ROUGE-L score of 28.55.
This research underscores the value of domain-specific training for optimizing
clinical diagnostic reasoning tasks.Comment: Accepted to the Proceedings of the 5th Clinical NLP Workshop at AC
Progress Note Understanding -- Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 Shared Task
Daily progress notes are common types in the electronic health record (EHR)
where healthcare providers document the patient's daily progress and treatment
plans. The EHR is designed to document all the care provided to patients, but
it also enables note bloat with extraneous information that distracts from the
diagnoses and treatment plans. Applications of natural language processing
(NLP) in the EHR is a growing field with the majority of methods in information
extraction. Few tasks use NLP methods for downstream diagnostic decision
support. We introduced the 2022 National NLP Clinical Challenge (N2C2) Track 3:
Progress Note Understanding - Assessment and Plan Reasoning as one step towards
a new suite of tasks. The Assessment and Plan Reasoning task focuses on the
most critical components of progress notes, Assessment and Plan subsections
where health problems and diagnoses are contained. The goal of the task was to
develop and evaluate NLP systems that automatically predict causal relations
between the overall status of the patient contained in the Assessment section
and its relation to each component of the Plan section which contains the
diagnoses and treatment plans. The goal of the task was to identify and
prioritize diagnoses as the first steps in diagnostic decision support to find
the most relevant information in long documents like daily progress notes. We
present the results of 2022 n2c2 Track 3 and provide a description of the data,
evaluation, participation and system performance.Comment: To appear in Journal of Biomedical Informatic
DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing
The meaningful use of electronic health records (EHR) continues to progress
in the digital era with clinical decision support systems augmented by
artificial intelligence. A priority in improving provider experience is to
overcome information overload and reduce the cognitive burden so fewer medical
errors and cognitive biases are introduced during patient care. One major type
of medical error is diagnostic error due to systematic or predictable errors in
judgment that rely on heuristics. The potential for clinical natural language
processing (cNLP) to model diagnostic reasoning in humans with forward
reasoning from data to diagnosis and potentially reduce the cognitive burden
and medical error has not been investigated. Existing tasks to advance the
science in cNLP have largely focused on information extraction and named entity
recognition through classification tasks. We introduce a novel suite of tasks
coined as Diagnostic Reasoning Benchmarks, DR.BENCH, as a new benchmark for
developing and evaluating cNLP models with clinical diagnostic reasoning
ability. The suite includes six tasks from ten publicly available datasets
addressing clinical text understanding, medical knowledge reasoning, and
diagnosis generation. DR.BENCH is the first clinical suite of tasks designed to
be a natural language generation framework to evaluate pre-trained language
models. Experiments with state-of-the-art pre-trained generative language
models using large general domain models and models that were continually
trained on a medical corpus demonstrate opportunities for improvement when
evaluated in DR. BENCH. We share DR. BENCH as a publicly available GitLab
repository with a systematic approach to load and evaluate models for the cNLP
community.Comment: Under revie
Recommended from our members
A common type system for clinical natural language processing
Background: One challenge in reusing clinical data stored in electronic medical records is that these data are heterogenous. Clinical Natural Language Processing (NLP) plays an important role in transforming information in clinical text to a standard representation that is comparable and interoperable. Information may be processed and shared when a type system specifies the allowable data structures. Therefore, we aim to define a common type system for clinical NLP that enables interoperability between structured and unstructured data generated in different clinical settings. Results: We describe a common type system for clinical NLP that has an end target of deep semantics based on Clinical Element Models (CEMs), thus interoperating with structured data and accommodating diverse NLP approaches. The type system has been implemented in UIMA (Unstructured Information Management Architecture) and is fully functional in a popular open-source clinical NLP system, cTAKES (clinical Text Analysis and Knowledge Extraction System) versions 2.0 and later. Conclusions: We have created a type system that targets deep semantics, thereby allowing for NLP systems to encapsulate knowledge from text and share it alongside heterogenous clinical data sources. Rather than surface semantics that are typically the end product of NLP algorithms, CEM-based semantics explicitly build in deep clinical semantics as the point of interoperability with more structured data types
Recommended from our members
Towards comprehensive syntactic and semantic annotations of the clinical narrative
Objective: To create annotated clinical narratives with layers of syntactic and semantic labels to facilitate advances in clinical natural language processing (NLP). To develop NLP algorithms and open source components. Methods: Manual annotation of a clinical narrative corpus of 127 606 tokens following the Treebank schema for syntactic information, PropBank schema for predicate-argument structures, and the Unified Medical Language System (UMLS) schema for semantic information. NLP components were developed. Results: The final corpus consists of 13 091 sentences containing 1772 distinct predicate lemmas. Of the 766 newly created PropBank frames, 74 are verbs. There are 28 539 named entity (NE) annotations spread over 15 UMLS semantic groups, one UMLS semantic type, and the Person semantic category. The most frequent annotations belong to the UMLS semantic groups of Procedures (15.71%), Disorders (14.74%), Concepts and Ideas (15.10%), Anatomy (12.80%), Chemicals and Drugs (7.49%), and the UMLS semantic type of Sign or Symptom (12.46%). Inter-annotator agreement results: Treebank (0.926), PropBank (0.891–0.931), NE (0.697–0.750). The part-of-speech tagger, constituency parser, dependency parser, and semantic role labeler are built from the corpus and released open source. A significant limitation uncovered by this project is the need for the NLP community to develop a widely agreed-upon schema for the annotation of clinical concepts and their relations. Conclusions: This project takes a foundational step towards bringing the field of clinical NLP up to par with NLP in the general domain. The corpus creation and NLP components provide a resource for research and application development that would have been previously impossible
- …